Selectively pairing an application presented in virtual space with a physical display

ABSTRACT

A method pairs content from a specific area in virtual space with a graphics driver for a physical display. An augmented reality device defines a virtual space, which is delimited by a spatial border that is created by the augmented reality device and is visible only to a user of the augmented reality device. One or more processors associate the virtual space with a graphics driver from a plurality of graphics drivers. A signal indicating a virtual space selection gesture is generated by the augmented reality device in response to the user making a physical motion that selects the virtual space, and sent to a video adapter hardware card. In response to receiving the signal indicating the virtual space selection gesture, the video adapter hardware card implements the graphics driver to display content associated with the virtual space on the physical display.

BACKGROUND

The present disclosure relates to the field of computers, andspecifically to computer displays. Still more particularly, the presentdisclosure relates to populating a physical computer display with anapplication that is presented by augmented reality glasses in virtualspace.

SUMMARY

In an embodiment of the present invention, a method pairs content from aspecific area in virtual space with a graphics driver for a physicaldisplay. An augmented reality device defines a virtual space, where thevirtual space is delimited by a spatial border that is created by theaugmented reality device, and where the spatial border is visible onlyto a user of the augmented reality device. One or more processorsassociate the virtual space with a graphics driver from a plurality ofgraphics drivers, where the graphics drivers generate images on aphysical display. A signal indicating a virtual space selection gestureis received from the augmented reality device, where the signalindicating the virtual space selection gesture is generated by theaugmented reality device in response to the user making a physicalmotion that selects the virtual space. In response to receiving a signalindicating the virtual space selection gesture, a hardware graphics cardimplements the graphics driver to display content associated with thevirtual space on the physical display.

In an embodiment of the present invention, a system pairs content from aspecific area in virtual space with a graphics driver for a physicaldisplay. An augmented reality device defines a virtual space, where thevirtual space is delimited by a spatial border that is created by theaugmented reality device, and where the spatial border is visible onlyto a user of the augmented reality device. One or more processorsassociate the virtual space with a particular graphics driver from aplurality of graphics drivers, wherein the graphics drivers generateimages on a physical display, where the one or more processors receive,from the augmented reality device, a signal indicating a virtual spaceselection gesture from the user of the augmented reality device, andwhere the signal indicating the virtual space selection gesture isgenerated by the augmented reality device in response to the user makinga physical motion that selects the virtual space. A hardware graphicscard, in response to receiving a signal indicating the virtual spaceselection gesture, implements the particular graphics driver to displaycontent associated with the virtual space on the physical display.

In an embodiment of the present invention, a computer program productpairs content from a specific area in virtual space with a graphicsdriver for a physical display. The computer program product includes acomputer readable storage medium having program code embodied therewith,and the computer readable storage medium is not a transitory signal perse. The program code is readable and executable by a processor toperform a method that includes defining a virtual space, where thevirtual space is delimited by a spatial border that is created by anaugmented reality device, and the spatial border is visible only to auser of the augmented reality device. The virtual space is associatedwith a particular graphics driver from a plurality of graphics drivers,where the graphics drivers generate images on a physical display. Asignal indicating the virtual space selection gesture is received fromaugmented reality device, where the signal indicating the virtual spaceselection gesture is generated by the augmented reality device inresponse to the user making a physical motion that selects the virtualspace. In response to receiving the signal indicating virtual spaceselection gesture, a hardware graphics card, implementing the particulargraphics driver, displays content associated with the virtual space onthe physical display.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an exemplary system and network in which the presentdisclosure may be implemented;

FIG. 2 illustrates an augmented reality device being used to selectivelydisplay virtual images on a physical display;

FIG. 3 depicts additional detail of the augmented reality device shownin FIG. 2;

FIG. 4 illustrates an exemplary view of a physical display surrounded byvirtual images as seen by a wearer of the augmented reality device shownin FIG. 2;

FIG. 5 depicts another exemplary view of a physical display surroundedby virtual images as seen by a wearer of the augmented reality deviceshown in FIG. 2;

FIG. 6 illustrates a virtual image being adjusted in size to conformwith apparent dimensions of a physical display; and

FIG. 7 is a high level flow-chart of one or more operations performed byone or more hardware devices to pair content from a specific area invirtual space with a graphics driver for a physical display.

DETAILED DESCRIPTION

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Java, Smalltalk, C++ or the like,and conventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

With reference now to the figures, and in particular to FIG. 1, there isdepicted a block diagram of an exemplary system and network that may beutilized by and/or in the implementation of the present invention. Notethat some or all of the exemplary architecture, including both depictedhardware and software, shown for and within computer 102 may be utilizedby software deploying server 150 and/or augmented reality device 152.

Exemplary computer 102 includes a processor 104 that is coupled to asystem bus 106. Processor 104 may utilize one or more processors, eachof which has one or more processor cores. A video adapter 108, whichdrives/supports a display 110, is also coupled to system bus 106. In oneor more embodiments of the present invention, video adapter 108 is ahardware video card. System bus 106 is coupled via a bus bridge 112 toan input/output (I/O) bus 114. An I/O interface 116 is coupled to I/Obus 114. I/O interface 116 affords communication with various I/Odevices, including a keyboard 118, a mouse 120, a media tray 122 (whichmay include storage devices such as CD-ROM drives, multi-mediainterfaces, etc.), a transceiver 124, and external USB port(s) 126.While the format of the ports connected to I/O interface 116 may be anyknown to those skilled in the art of computer architecture, in oneembodiment some or all of these ports are universal serial bus (USB)ports.

Associated with (if not part of) the video adapter 108 is a graphicsdriver(s) 109. Graphics driver(s) 109, which in one embodiment is partof operating system 138, is software that controls how the video adapter108 (e.g., a video card) generates text and images on the display 110.In an embodiment of the present invention, multiple graphics drivers 109can be associated with the video adapter 108. In this embodiment, eachof the multiple graphics drivers 109 are associated with a particulararea of virtual space and/or with a particular application, as describedbelow.

As depicted, computer 102 is able to communicate with a softwaredeploying server 150, using a network interface 130. Network interface130 is a hardware network interface, such as a network interface card(NIC), etc. Network 128 may be an external network such as the Internet,or an internal network such as an Ethernet or a virtual private network(VPN).

A hard drive interface 132 is also coupled to system bus 106. Hard driveinterface 132 interfaces with a hard drive 134. In one embodiment, harddrive 134 populates a system memory 136, which is also coupled to systembus 106. System memory is defined as a lowest level of volatile memoryin computer 102. This volatile memory includes additional higher levelsof volatile memory (not shown), including, but not limited to, cachememory, registers and buffers. Data that populates system memory 136includes computer 102's operating system (OS) 138 and applicationprograms 144.

OS 138 includes a shell 140, for providing transparent user access toresources such as application programs 144. Generally, shell 140 is aprogram that provides an interpreter and an interface between the userand the operating system. More specifically, shell 140 executes commandsthat are entered into a command line user interface or from a file.Thus, shell 140, also called a command processor, is generally thehighest level of the operating system software hierarchy and serves as acommand interpreter. The shell provides a system prompt, interpretscommands entered by keyboard, mouse, or other user input media, andsends the interpreted command(s) to the appropriate lower levels of theoperating system (e.g., a kernel 142) for processing. Note that whileshell 140 is a text-based, line-oriented user interface, the presentinvention will equally well support other user interface modes, such asgraphical, voice, gestural, etc.

As depicted, OS 138 also includes kernel 142, which includes lowerlevels of functionality for OS 138, including providing essentialservices required by other parts of OS 138 and application programs 144,including memory management, process and task management, diskmanagement, and mouse and keyboard management.

Application programs 144 include a renderer, shown in exemplary manneras a browser 146. Browser 146 includes program modules and instructionsenabling a world wide web (WWW) client (i.e., computer 102) to send andreceive network messages to the Internet using hypertext transferprotocol (HTTP) messaging, thus enabling communication with softwaredeploying server 150 and other computer systems.

Application programs 144 in computer 102's system memory (as well assoftware deploying server 150's system memory) also include an augmentedreality coordination logic (ARCL) 148. ARCL 148 includes code forimplementing the processes described below, including those described inFIGS. 2-7. In one embodiment, computer 102 is able to download ARCL 148from software deploying server 150, including in an on-demand basis,wherein the code in ARCL 148 is not downloaded until needed forexecution. Note further that, in one embodiment of the presentinvention, software deploying server 150 performs all of the functionsassociated with the present invention (including execution of ARCL 148),thus freeing computer 102 from having to use its own internal computingresources to execute ARCL 148.

Transceiver 124 (and/or any input/output (I/O) interface such as USBport(s) 126) is able to electronically communicate with an augmentedreality device 152, such as a pair of augmented reality (AR) glasses. ARglasses are a wearable electronic device that allows a wearer to seephysical reality with an overlaid virtual reality (e.g., a heads-updisplay). Thus, the wearer/user is able to see both what is real (i.e.,in real physical space) as well as what is virtual (i.e., generated by acomputer and displayed on the heads-up display), thereby “augmenting”reality (that is real) with computer-generated images (which arevirtual).

Note that the hardware elements depicted in computer 102 are notintended to be exhaustive, but rather are representative to highlightessential components required by the present invention. For instance,computer 102 may include alternate memory storage devices such asmagnetic cassettes, digital versatile disks (DVDs), Bernoullicartridges, and the like. These and other variations are intended to bewithin the spirit and scope of the present invention.

With reference now to FIG. 2, an augmented reality device 200 being usedto selectively display virtual images on a physical display ispresented. As shown in FIG. 2, a user 204 (e.g., the wearer of theaugmented reality device 200) is able to “see” two types of images.

The first type of images seen by the user 204 are of real objects, whichare viewed through a transparent portion of the augmented reality device200. An example of such a real object is the physical display 210(analogous to the display 110 in FIG. 1) and the content (e.g.,application, webpage, etc.) being presented on the physical display bythe computer 202 (analogous to the computer 102 in FIG. 1).

The second type of images seen by the user 204 are of virtual images,which are shown to the user on a display screen within the augmentedreality device 200. As described herein, the virtual images may beoverlaid onto the transparent portion of the augmented reality device200 through which the user 204 sees the real world. Thus, these overlaidvirtual images “augment” reality. Examples of virtual images visible tothe user 204 include, but are not limited to, virtual applicationdisplays, virtual webpages, virtual video clips, etc., all of which aregenerated by the augmented reality device 200.

As shown in FIG. 2, within computer 202 are graphics drivers 209(analogous to the graphics drivers 109 shown in FIG. 1). Graphicsdrivers 209 are defined as software units that generate images on aphysical display (e.g., physical display 210). In an embodiment of thepresent invention, each of the graphics drivers 209 is associated withone, and only one, of the applications 244 (such as a spreadsheet, avideo application, a word processing application, a browser, etc.).Furthermore and in one embodiment of the present invention, fullversions of the applications 244 and their associated graphics driversare not available to the augmented reality device 200. Rather, only astatic descriptor image of the applications 244 is displayed on theaugmented reality device 200. However, in another embodiment of thepresent invention, full versions of the applications 244 and theirassociated graphics drivers are available to the augmented realitydevice 200, such that real-time dynamic (changeable) content for theapplications 244 is available for display on the augmented realitydevice 200.

With reference now to FIG. 3, additional detail of the augmented realitydevice 200 shown in FIG. 2 in accordance with one or more embodiments ofthe present invention is presented. As described above, augmentedreality device 200 includes a transparent viewing glass 302, throughwhich the wearer of the augmented reality device 200 is able to view thereal (physical) world. However, in order to augment this view of thereal world with virtual images, an overlay projector 304 is able toproject images onto the transparent viewing glass 302, such that theuser/viewer/wearer of the augmented reality device 200 sees both realphysical objects from the world as well as the augmented virtual images.Thus, as shown in FIG. 2, the user 204 is able to see both the physicaldisplay 210 (and content from the applications 244 that is generatedthereon) as well as virtual images, such as those in a virtual space208.

Virtual space 208 may be two-dimensional or three-dimensional. Forexample and as shown in FIG. 3, a two-dimensional (2-D) spatial border306 delimits only two dimensions. Therefore, when the user moves hishand 206 (as shown in FIG. 2) through space, any area in real space thatis “behind” the 2-D spatial border 306 as viewed by the user 204 isconsidered by the system (e.g., computer 202) to be part of the virtualspace 208 being selected by the user 204. However, when a user moves hishand 206 through the confined three-dimensional (3-D) space that isdefined by 3-D spatial border 308, then the system only performs actionsassociated with that confined 3-D space. This 3-D feature allows variousvirtual spaces to occupy a same line of sight of the user 204, where onevirtual space is in front of (or behind) another virtual space.

Thus, the 3-D spatial border 308 delimits a particular volume of realspace as a virtual space. The user 204 “sees” the 3-D spatial border 308“floating” in real space due to the optical illusion created by theimage projected by the overlay project 304 onto the transparent viewingglass 302. When the user 204 moves his hand 206 (or any other object,such as a wand, a glove, a pointer, etc.) through the virtual space 208created by the 3-D spatial border 308, then the computer 202 respondswith an action that is associated with 1) the particular virtual space208, and 2) the gesture/movement performed by the user's hand 206.

Returning to FIG. 3, the 3-D position of the augmented reality device200 is needed to clarify/identify 1) where the user 204 is looking, andthus 2) which virtual space is being engaged with by the user 204.

In an embodiment of the present invention, the 3-D position of theaugmented reality device 200 is established by 3-D accelerometers 310,which detect a 3-D movement of the augmented reality device 200. Variouselectronic devices can be used to detect acceleration movement in one ormore axis, including piezoelectric, piezoresistive, and capacitivecomponents that convert mechanical motion into electrical signals. Inone embodiment the 3-D accelerometer(s) 310 incorporates the use ofmicro electro-mechanical systems (MEMS), which measure deflection of amicro-lever that is counter-weighted with a known mass.

In one embodiment of the present invention, the 3-D position of theaugmented reality device 200 is determined by electromyographic (EMG)sensors 312 shown in FIG. 3. EMG sensors 312 are sensors that detectmuscular movements by the user 204 shown in FIG. 2. That is, whenever awearer of the augmented reality device 200 moves a muscle in his/herhead, an electrical signal (i.e., an ion-induced change in electricalpotential across the nerve membrane) is transmitted down axons (nervefibers) that control the muscles. A motor neuron and the muscles that itinnervates are collectively known as a motor unit. When the motor neuroncauses the muscles in the motor unit to contract, electrical activity isstimulated in these muscles. The sum of the electrical activity from themotor neuron and the muscles in the motor unit create a motor unitaction potential (MUAP), which is measured by the EMG sensors 312. Themeasured MUAP is analyzed by the processing logic (e.g., microprocessor314 shown in FIG. 3) to identify the level of muscle movement. That is,electrical signals are specific enough to identify a location of whichmuscles are being moved. Whenever the wearer moves his head, a uniqueMUAP pattern occurs. Thus, the MUAP pattern detected in real time by theEMG sensors 312 is compared with known/stored MUAP readings for specifichead positions. This comparison leads to a determination of where theaugmented reality device 200 is in 3-D real space, and thus wherein thegenerated virtual space 208 is located.

In order to determine the location of the user's hand 206 shown in FIG.2, various devices within the augmented reality device 200 can be used.

For example and in one embodiment of the present invention, motionsensors 316 detect movement of the user's hand 206 and the location ofthe user's hand 206. Motion sensors 316 use optical, microwave,acoustic, and/or infrared sensors that sense a signal being sent fromthe user's hand 206. For example, light or infrared transmitters on aglove (not shown) worn on the user's hand 206 can be detected by themotion sensors 316 to determine where the user 204 is moving his hand.Alternatively, acoustic signals can be transmitted from the motionsensors 316 and bounced back to the motion sensors 316. By using thedirection of the bounced-back acoustic signals and the time durationbetween transmitting and receiving the acoustic signals, the augmentedreality device 200 is able to determine the 3-D location of the user'shand 206. Similarly, electromagnetic radiation (e.g., infrared light,visible light, radio frequency signals, etc.) can be transmitted fromthe motion sensors 316 and bounced back to the motion sensors 316. Aswith the use of acoustic signals, by using the direction of thebounced-back electromagnetic radiation and the time duration betweentransmitting and receiving the electromagnetic radiation, the augmentedreality device 200 is able to determine the 3-D location of the user'shand 206.

In one embodiment of the present invention, the 3-D position of theuser's hand 206 is detected by passive infrared (IR) sensors 318. Theuser's hand 206 generates heat due to normal human physiology. Bydetecting the strength and direction of IR radiation, the IR sensors 318provide enough information/data to the microprocessor 314 to determinethe 3-D position of the user's hand 206, by comparing the IR sensorreadings with known table/charts.

In one embodiment of the present invention, the 3-D position of theuser's hand 206 is detected by camera(s) 320. The camera(s) 320 capturea video image of the user's hand 206. By comparing the size of thecaptured image to a known size of the user's hand 206, the distance fromthe augmented reality device 200 to the user's hand 206 is determined bythe microprocessor 314. The direction of the user's hand 206 relative tothe position of the augmented reality device 200 can be determined byother sensors (e.g., 3-D accelerometers 310) within the augmentedreality device 200.

FIG. 3 also depicts an input/output (I/O) card 322, which allows the I/Ocard 322 to exchange data/information with other devices, includingcomputer 202 shown in FIG. 2, electronic devices held by the user 204,etc.

Referring now to FIG. 4, an exemplary view (as seen by the wearer of theaugmented reality device 200 shown in FIG. 2) through a transparentviewing glass 302 is shown. As depicted in FIG. 4, the user/wearer ofthe augmented reality device 200 sees both real and virtual objects. Thereal object is the physical display 410 (similar to the physical display210 shown in FIG. 2). The virtual objects are in the depicted multiplevirtual spaces 408 a-408 c (where “c” is any integer), which areanalogous to the virtual space 208 shown in FIG. 2. Each of the virtualspaces 408 a-408 c is delimited by boundaries generated by the augmentedreality device 200. In one or more embodiments of the present invention,each of the virtual spaces 408 a-408 c is associated with a particulargraphic driver (e.g., from graphics drivers 209 in FIG. 2) and/or aparticular application (e.g., from applications 244 in FIG. 2). That is,in one embodiment, the overlay projects 304 in FIG. 3 projects imagesgenerated by the applications 244 onto the transparent viewing glass302, thereby creating an appearance of these images within the virtualspaces 408 a-408 c. In one embodiment, each virtual space from thevirtual spaces 408 a-408 c provides an environment for displayingcontent from a particular application. In one embodiment, this contentis a full version of what the application creates, while in anotherembodiment the virtual spaces 408 a-408 c only provide an environmentfor an icon for the applications.

For example, assume that virtual space 408 a is associated with awebpage. In one embodiment, this webpage is visible to a wearer of theaugmented reality device 200 (via the transparent viewing glass 302) ina real-time active mode. That is, whatever would appear on a realmonitor also appears on the virtual image shown in virtual space 408 a.However, in another embodiment, only an icon, snapshot, or other staticdisplay for that webpage appears in virtual space 408 a.

In either embodiment (whether the dynamic webpage or just an iconicimage for the webpage appears in the virtual space 408 a), the presentinvention allows the wearer of the augmented reality device 200 to movethe webpage from virtual space (e.g., within virtual space 408 a) toreal space (e.g., onto the physical display 410). Various actions may betaken by the user (e.g., sweeping his/her hand through virtual space 408a towards the physical display 410) which will cause the webpage to bedisplayed on the physical display 410. This movement of the display ofthe webpage from virtual space 408 a to the physical display 410 furtherresults in the computer 202 controlling not only the display of thewebpage on the physical display 210/410, but also enables the use of akeyboard or other input (e.g., keyboard 118 or mouse 120 shown inFIG. 1) to control/manipulate the webpage and its display.

That is, the present invention allows a user to move a virtual imageassociated with an application onto a physical display, thereby enablingthe user to 1) see the content of the application on the physicaldisplay, and 2) have the ability to manipulate the application and itscontent using hardware associated with the computer that supports thephysical display.

Referring now to FIG. 5, assume that the physical display 210 shown inFIG. 2 is virtually surrounded by four virtual spaces, described asvirtual graphical devices 1-4 (virtual graphical devices 501 a-501 d).Note that each of the virtual graphical devices 501 a-501 d has a samevirtual dimension as the physical display 210 itself. That is, thedimensions of what the wearer of the augmented reality device 200 seeson the transparent viewing glass 302 for each of the virtual graphicaldevices 501 a-501 d has been sized to match the dimensions of thephysical display 410 seen though the transparent viewing glass 302.

In one embodiment of the present invention, the apparent dimensions ofthe physical display 210 are derived from physical markers found on thephysical display 210. For example, consider marker 503 in FIG. 5, whichis one of four markers (shown as triangles) positioned at each of thefour corners of the physical display 210. The triangles/markers may bemounted on a physical housing of the physical display 210, or they maybe electronically displayed on the screen of the physical display 210.In either embodiment, the camera 320 in FIG. 3 will take a photograph ofthe physical display 210, including the four triangles/markers. Themicroprocessor 314 then runs an edge detection algorithm on the capturedimage to identify the four triangles/markers. The location of the fourtriangles/markers allows the microprocessor 314 to determine theapparent size of the physical display 210 the user's perspective. Oncethe size and position of the physical display 210 are ascertained, anaugmented reality (AR) marker 505, which surrounds the physical display210 in virtual reality, denotes the screen position and orientation ofthe physical display 210 as a base/reference position, from which thevirtual graphical devices 501 a-501 d are positioned.

While the examples shown in FIG. 4 and FIG. 5 assume that the physicaldisplay 210 is normal to (i.e., perpendicular to) a line of sight of thewearer of the augmented reality device 200, at times the physicaldisplay will be tilted towards or away from the user/wearer,particularly if the physical display 210 is part of a tablet computer orother movable device. In order to compensate for such tilting, thesystem utilizes a compensation procedure.

Thus, in FIG. 6, assume that a physical display 210 has a height shownby dashed line 609. However, if the physical display 210 is tiltedtowards the viewer 604, then the viewer 604, looking along line of sight603, will perceive the physical display 210 as having a height of onlythat shown by line 601, due to the physical display 210 now being tiltedalong line 610. (Note that the length of line 610 and the length of line609 are the same.)

Thus, the perceived differences in height between line 610 and line 609are merely optical illusions causes by the tilting of the physicaldisplay 210.

To calculate the angle of orientation, the law of cosines is used,where:

${\cos\; A} = {\frac{adjacent}{hypotenuse} = {\frac{b}{h}.}}$

Thus, the cosine of A (angle 607) is equal to the length of line 609 upto line 611, divided by the length of line 610. The length of theadjacent side is the same as that for line 601, which is measured by aphotograph of the tilted physical display 210 taken by the camera 320 inthe augmented reality device 200. The length of the hypotenuse is theactual height of the physical display 210, and can be derived by thecamera 320 taking a photo of the physical display 210 when perpendicularto the line of sight 603, or it may be provided by an identifiertransmitted from the physical display (e.g., a universal uniqueidentifier—UUID), which contains the dimensions of the physical display.This information thus leads to the value of angle A (angle 607),enabling the system to adjust the size of the virtual graphical devices501 a-501 d shown in FIG. 5.

Referring now to FIG. 7, a high level flow-chart of one or moreoperations performed by one or more hardware devices to pair contentfrom a specific area in virtual space with a graphics driver for aphysical display is presented.

After initiator block 702, an augmented reality device (e.g., augmentedreality device 200 in FIG. 2), defines a first virtual space (e.g.,virtual space 208 in FIG. 2), as described in block 704. The firstvirtual space is delimited by a first spatial border (e.g., 3-D spatialborder 308 in FIG. 3) that is created by the augmented reality device.The first spatial border is visible only to a user of the augmentedreality device, as it is displayed on the transparent viewing glass 302in FIG. 3. In an alternate embodiment, spatial borders and theirassociated virtual graphical devices can be transmitted to a seconddevice, such as a second video display, thus allowing multiple personsto see the augmented reality (generated by the overlay projector 304 inFIG. 3) and/or what the user sees from the real physical world (ascaptured by the camera(s) 320 in FIG. 3).

With reference now to block 706, one or more processors (e.g.,microprocessor 314 in FIG. 3 or processor 104 in FIG. 1) then associatethe first virtual space with a first graphics driver from a plurality ofgraphics drivers, wherein the graphics drivers generate images on aphysical display. That is, the processors assign a particular virtualspace (e.g., virtual space 208) with a particular graphic driver (fromgraphics drivers 209 in FIG. 2). Thus, each virtual space is specificfor a particular application/content and the graphics driver thatsupports that particular application/content.

With reference now to block 708, a video adapter hardware card (e.g.,video adapter 108 in FIG. 1) receives a first signal indicating a firstvirtual space selection gesture from the augmented reality device. Thefirst signal indicating the first virtual space selection gesture isgenerated by the augmented reality device in response to the user makinga first physical motion that selects the first virtual space, such asmoving his/her hand through the first virtual space (i.e., the area inreal space that “appears” to be bounded by virtual boundaries to theuser).

As described in block 710, in response to the video adapter hardwarecard receiving the first signal indicating the first virtual spaceselection gesture, the video adapter hardware card, by implementing thefirst graphics driver, displays content associated with the firstvirtual space on the physical display. That is, what earlier was onlyviewable through the virtual augmented device is not placed on thephysical device, such that it can be directly manipulated by thecomputer that supports that physical device.

The flow-chart ends at terminator block 712.

In an embodiment of the present invention, the first virtual space isfrom multiple virtual spaces that are defined by the augmented realitydevice, and the multiple virtual spaces are visible only to the user ofthe augmented reality device. That is, multiple virtual spaces can beviewable with the use of the augmented reality device (see FIG. 4 andFIG. 5).

In an embodiment of the present invention, real-time content beingdisplayed on the physical display is not displayed on the augmentedreality device. That is, in this embodiment, when content is beingdisplayed on the physical display, it no longer is visible in virtualspace that is created by the augmented reality device. Rather, thesystem deletes any virtual space that was previously devoted to thecontent that is now being displayed on the physical display.

As described herein, in one embodiment of the present invention, thefirst virtual space selection gesture is a movement of the user throughthe first virtual space. For example, the user may move his/her handthrough the region in space that correlates to the first virtual spacebeing displayed on the augmented reality device. However, in anotherembodiment, merely aiming the augmented reality device towards aparticular virtual space is enough to select the content being presentedin that virtual space for display on the physical display. Thus, theuser may move his/her head such that the particular virtual space iscentered in the transparent viewing glass 302 shown in FIG. 3. By takingsome secondary act (such as blinking one's eyes, or simply keeping theaugmented reality device in a fixed position for longer than somepredetermined amount of time—e.g., three seconds), then the system willselect the virtual space and its associated application/driver/contentfor display on the physical display. Determining where the user islooking and which virtual space is being looked at are achieved by thecomponents of the augmented reality device 200 described herein,particularly in FIG. 3.

In an embodiment of the present invention, one or more processors, usingsensor readings from the augmented reality, adjust a dimension of thefirst virtual space according to a deflection, in real physical space,of the physical display, as described in FIG. 6.

In an embodiment of the present invention, the augmented reality devicedefines a second virtual space, which is delimited by a second spatialborder that is created by the augmented reality device, and where thesecond spatial border is visible only to the user of the augmentedreality device. One or more processors then associate the second virtualspace with a second graphics driver from the plurality of graphicsdrivers. The video adapter hardware card receives a second signalindicating a second virtual space selection gesture from the augmentedreality device, where the second signal indicates the second virtualspace selection gesture that is generated by the augmented realitydevice in response to the user making a second physical motion thatselects the second virtual space. In response to the video adapterhardware card receiving the second signal indicating the second virtualspace selection gesture, the video adapter hardware card replaces thefirst graphics driver with the second graphic driver, such that contentassociated with the second virtual space is now on the physical display,and content that was initially associated with the first virtual spaceand moved to the physical display is now removed from the physicaldisplay. That is, a second virtual space can evict content, which wasinitially associated with a first virtual space, from the physicaldisplay.

In an embodiment of the present invention, the hardware graphics carddisplays multiple positioning icons on the physical display (se marker503 and the other positioning triangles in FIG. 5). As described in FIG.5, one or more processors are able to calculate a perceived size of thephysical display based on the multiple positioning icons on the physicaldisplay. The augmented reality device is then able to adjust a size ofthe first virtual space to match the perceived size of the physicaldisplay.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the presentinvention. As used herein, the singular forms “a”, “an” and “the” areintended to include the plural forms as well, unless the context clearlyindicates otherwise. It will be further understood that the terms“comprises” and/or “comprising,” when used in this specification,specify the presence of stated features, integers, steps, operations,elements, and/or components, but do not preclude the presence oraddition of one or more other features, integers, steps, operations,elements, components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of various embodiments of the present invention has beenpresented for purposes of illustration and description, but is notintended to be exhaustive or limited to the present invention in theform disclosed. Many modifications and variations will be apparent tothose of ordinary skill in the art without departing from the scope andspirit of the present invention. The embodiment was chosen and describedin order to best explain the principles of the present invention and thepractical application, and to enable others of ordinary skill in the artto understand the present invention for various embodiments with variousmodifications as are suited to the particular use contemplated.

Note further that any methods described in the present disclosure may beimplemented through the use of a VHDL (VHSIC Hardware DescriptionLanguage) program and a VHDL chip. VHDL is an exemplary design-entrylanguage for Field Programmable Gate Arrays (FPGAs), ApplicationSpecific Integrated Circuits (ASICs), and other similar electronicdevices. Thus, any software-implemented method described herein may beemulated by a hardware-based VHDL program, which is then applied to aVHDL chip, such as a FPGA.

Having thus described embodiments of the present invention of thepresent application in detail and by reference to illustrativeembodiments thereof, it will be apparent that modifications andvariations are possible without departing from the scope of the presentinvention defined in the appended claims.

What is claimed is:
 1. A system for pairing content from a specific areain virtual space with a graphics driver for a physical display, thesystem comprising: an augmented reality device, wherein the augmentedreality device defines a first virtual space, wherein the first virtualspace is delimited by a first spatial border that is created by theaugmented reality device, and wherein the first spatial border isvisible only to a user of the augmented reality device; one or moreprocessors that associate the first virtual space with a first graphicsdriver from a plurality of graphics drivers, wherein the graphicsdrivers generate images on a physical display, wherein the physicaldisplay is external to the augmented reality device; and a video adapterhardware card, wherein the video adapter hardware card, in response toreceiving a first signal indicating a selection gesture performed by theuser within the first virtual space to select an object within the firstvirtual space, implements the first graphics driver to display theobject associated with the first virtual space on the physical displaythat is external to the augmented reality device, wherein the selectiongesture is performed by the user swiping his/her hand from the objectwithin the first virtual space towards the physical display to cause theobject to be displayed on the physical display, wherein the physicaldisplay has a universal unique identifier (UUID) that contains physicaldimensions of the physical display, and wherein the one or moreprocessors: receive the UUID from the physical display; and adjust,based on the physical dimensions described in the UUID received from thephysical display, a dimension of the first virtual space according to adeflection, in real physical space, of the physical display, wherein thedimension of the first virtual space perceived by the user of theaugmented reality device is adjusted to match actual physical dimensionsof the physical display such that a virtual graphical device within thefirst virtual space has a same set of perceived dimensions as the actualphysical dimensions of the physical display.
 2. The system of claim 1,wherein the first virtual space is from multiple virtual spaces that aredefined by the augmented reality device, wherein the multiple virtualspaces are visible only to the user of the augmented reality device, andwherein real-time content being displayed in the multiple virtual spacesis visible without being displayed on the physical display.
 3. Acomputer program product for pairing content from a specific area invirtual space with a graphics driver for a physical display, thecomputer program product comprising a computer readable storage mediumhaving program code embodied therewith, wherein the computer readablestorage medium is not a transitory signal per se, and wherein theprogram code is readable and executable by a processor to perform amethod comprising: defining a virtual space, wherein the virtual spaceis delimited by a spatial border that is created by an augmented realitydevice, and wherein the spatial border is visible only to a user of theaugmented reality device; associating the virtual space with a specificgraphics driver from a plurality of graphics drivers, wherein thegraphics drivers generate images on a physical display, wherein thephysical display is external to the augmented reality device; receiving,by a video adapter hardware card, a virtual space selection signal fromthe augmented reality device, wherein the virtual space selection signalis generated by the augmented reality device in response to the usermaking a selection gesture that selects an object within the virtualspace, wherein the selection gesture is performed by the user swipinghis/her hand from the selected object within the virtual space towardsthe physical display to cause the selected object to be displayed on thephysical display; and in response to receiving the virtual spaceselection signal, displaying, by the video adapter hardware cardimplementing the specific graphics driver, the selected object on thephysical display that is external to the augmented reality device,wherein the physical display has a universal unique identifier (UUID)that contains physical dimensions of the physical display, and whereinthe method further comprises: receiving the UUID from the physicaldisplay; and adjusting, based on the physical dimensions described inthe UUID received from the physical display, a dimension of the virtualspace according to a deflection, in real physical space, of the physicaldisplay, wherein the dimension of the virtual space perceived by theuser of the augmented reality device is adjusted to match the actualphysical dimensions of the physical display such that a virtualgraphical device within the virtual space has a same set of perceiveddimensions as the actual physical dimensions of the physical display. 4.The computer program product of claim 3, wherein the virtual space isfrom multiple virtual spaces that are defined by the augmented realitydevice, wherein the multiple virtual spaces are visible only to the userof the augmented reality device, and wherein real-time content beingdisplayed in the multiple virtual spaces is visible without beingdisplayed on the physical display.
 5. The computer program product ofclaim 3, wherein the virtual space selection signal is a movement of theuser through the virtual space.
 6. The computer program product of claim3, wherein the method further comprises: defining, by the augmentedreality device, a second virtual space, wherein the second virtual spaceis delimited by a second spatial border that is created by the augmentedreality device, and wherein the second spatial border is visible only tothe user of the augmented reality device; associating the second virtualspace with a second graphics driver from the plurality of graphicsdrivers; receiving, by the video adapter hardware card, a second virtualspace selection signal from the augmented reality device, wherein thesecond virtual space selection signal is generated by the augmentedreality device in response to the user making a second selection gesturethat selects the second virtual space; and in response to the videoadapter hardware card receiving the second virtual space selectionsignal, displaying, by the video adapter hardware card replacing thefirst graphics driver with the second graphics driver, contentassociated with the second virtual space on the physical display,wherein content associated with the virtual space is removed from thephysical display.
 7. The computer program product of claim 3, whereinthe content associated with the virtual space and displayed on thephysical display is an application program.
 8. The computer programproduct of claim 3, wherein the content displayed on the physicaldisplay is a software application, and wherein the method furthercomprises: displaying an application icon in the virtual space, whereinthe application icon is a static descriptor image that describes thesoftware application without presenting a full version of the softwareapplication, and wherein the virtual space selection signal of theapplication icon causes the full version of the software application tobe displayed on the physical display.
 9. A computer program product forpairing content from a specific area in virtual space with a graphicsdriver for a physical display, the computer program product comprising acomputer readable storage medium having program code embodied therewith,wherein the computer readable storage medium is not a transitory signalper se, and wherein the program code is readable and executable by aprocessor to perform a method comprising: defining a virtual space,wherein the virtual space is delimited by a spatial border that iscreated by an augmented reality device, and wherein the spatial borderis visible only to a user of the augmented reality device; associatingthe virtual space with a specific graphics driver from a plurality ofgraphics drivers, wherein the graphics drivers generate images on aphysical display, wherein the physical display is external to theaugmented reality device; receiving, by a video adapter hardware card, avirtual space selection signal from the augmented reality device,wherein the virtual space selection signal is generated by the augmentedreality device in response to the user making a selection gesture thatselects an object within the virtual space, wherein the selectiongesture is performed by the user swiping his/her hand from the selectedobject within the virtual space towards the physical display to causethe selected object to be displayed on the physical display; in responseto receiving the virtual space selection signal, displaying, by thevideo adapter hardware card implementing the specific graphics driver,the selected object on the physical display that is external to theaugmented reality device, displaying, by a hardware graphics card,multiple positioning icons on the physical display; determining aperceived size of the physical display based on the multiple positioningicons on the physical display; and adjusting, by the augmented realitydevice, a size of the virtual space to match the perceived size of thephysical display.